Skip to content

[Bug] Match checkpoint directory names exactly - #4292

Open
taking-lying-flat wants to merge 1 commit into
pytorch:mainfrom
taking-lying-flat:fix/checkpoint-step-dir-parser
Open

[Bug] Match checkpoint directory names exactly#4292
taking-lying-flat wants to merge 1 commit into
pytorch:mainfrom
taking-lying-flat:fix/checkpoint-step-dir-parser

Conversation

@taking-lying-flat

@taking-lying-flat taking-lying-flat commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Problem

Checkpoint discovery and stale-checkpoint purging independently parse directory names with re.search using the step-(digits) pattern. Because re.search accepts a match anywhere in the string, names such as step-100.backup and foo-step-100 are treated as checkpoint step 100 even though they are outside the canonical step-N namespace.

This creates two distinct failure modes:

  1. During latest-checkpoint discovery, _find_load_step validates metadata under the original matched directory but retains only the captured integer. The loader then reconstructs the canonical path step-100. As a result, the directory that was validated can differ from the directory that is loaded, causing a missing-path failure or selecting a different checkpoint than the one discovery inspected.
  2. During retention cleanup, the purge path keeps the original matched filename and queues that exact path for deletion. A backup or otherwise unrelated directory containing a step-N substring can therefore be sorted as a stale checkpoint and deleted.

The root cause is substring matching combined with duplicated checkpoint-name parsing in discovery and purge.

Fix

Add one _parse_checkpoint_step helper using re.fullmatch with the step-(digits) pattern, and use it for both latest-checkpoint discovery and stale-checkpoint purging. Only a complete canonical directory name can now produce a step number. Latest loading already consumes the result of _find_load_step and reconstructs the same canonical step-N name, so parsing and path construction now agree.

The parser returns None for invalid names and uses explicit None checks so step-0 remains valid.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 23, 2026
@tianyu-l
tianyu-l requested a review from ivy-zhou August 23, 2026 04:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants